Method and system for storing packets for a bonded communication links

ABSTRACT

Method and system for storing packets received from a bonded communication links according to latency of the communication link that has the largest latency among all communication links of the bonded communication links. Embodiments of present inventions can be applied to bonded communication links, including wireless connection, Ethernet connection, Internet Protocol connection, asynchronous transfer mode, virtual private network, WiFi, high-speed downlink packet access, GPRS, LTE, and X.25. The present invention presents methods comprising the steps of estimating storage size of a queue, wherein the queue is for storage the one or more packets received from the bonded communication links. The storage size is based on one or more factors, including largest latency, bandwidth of each of the plurality of communication links, and allowed time duration of packet storage

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a non-provisional continuation-in-partapplication that claims the priority and benefits of and is based onU.S. application Ser. No. 13/822,637 titled “METHOD AND SYSTEM FORREDUCTION OF TIME VARIANCE OF PACKETS RECEIVED FROM BONDED COMMUNICATIONLINKS” filed on Jun. 20, 2013. The contents of the above-referencedapplication are herein incorporated by reference.

TECHNICAL FIELD

This invention relates in general to network communications and, moreparticularly, to a method and system for allocating storage for queuefor processing packets received from bonded communication linksaccording to latency difference among the bonded communication links andsequence numbers.

BACKGROUND ART

Network devices, such as routers, may be configured to distributeoutgoing traffic, which may be originated from an application within alocal area network or from a network device, across bonded communicationlinks associated with multiple egress interfaces, logic connections,network tunnels, virtual private networks and etc. There are a fewbonded communication links implementations, such as bonding, and PPPMultilink Protocol. Network traffic can be usually carried by packetsthrough wired or wireless and public or private networks through bondedcommunication links. In order to allow a destination network device(DND) to determine the sequence of the packets, it is a common practiceto assign a sequence number to each packet.

Each packet, when arriving at a DND, may experience different delay aseach of the bonded communication links may have different latency anddifferent amount of bandwidth available. Therefore packets may arrive atthe DND in a bursty fashion and out-of-sequence. Also, some of thepackets may never arrive at the DND because they are lost.

It is common that a DND may store the packets in a queue, which isimplemented in a memory, temporarily in order to reduce the possibilitythat the packets delivered are not in sequence. However, currentstate-of-art implementations of delivering packets received in a bondedcommunication links network results in large time-variance andout-of-sequence packet delivery even with implementation of a queue.Further, the storage of the queue needs to be allocated for storing thepackets. If the storage size is too large, some of computer resourcesmay be wasted. If the storage size is too small, packets may bediscarded too early.

Advantageous Effects

Network traffic received from bonded communication links are deliveredto a device, a network interface or a process of a destination networkdevice in sequence with higher probability and less time variancecomparing to a destination network device without implementing thisinvention while an estimated storage space is allocated for storingpackets.

SUMMARY OF THE INVENTION

The invention includes an implementation that reduces the time varianceof delivering packets to a device, a network interface or a process of adestination network device (DND) according to latency difference amongbonded communication links (Latency Difference). The sequence number(SEQ) of the packets received may also be used with latency differenceto reduce the time variance. It is a common knowledge that a sourcenetwork device (SND), which has the capabilities of distributing packetsacross bonded communication links, assigns consecutive SEQ to packetsbefore sending the packets to the bonded communication links.

The value of Latency Difference is based on the time difference ofpackets with consecutive SEQ arriving at the DND through the bondedcommunication links. The value of Latency Difference may change asnetwork conditions of bonded communication links change.

In one implementation, the DND delivers a packet without storing thepacket to a queue if the packet is arriving from the one of the bondedcommunication links which has the largest latency.

In one implementation, at the DND, an expected SEQ (E-SEQ) is calculatedbased on Latency Difference and SEQ of the previous packets sent to adevice, a network interface or a process of a destination networkdevice. When a packet arrives at the DND, the DND compares the SEQ ofthe packet (P-SEQ) against the E-SEQ. If P-SEQ is smaller than E-SEQ,the packet is then delivered without storing the packet into a queuebecause the packet has arrived at the DND later than expected. If thepacket arrives from one of the bonded communication links which has thelargest latency and its P-SEQ is larger than the E-SEQ, the packet isthen stored in a queue for later delivery because the packet is arrivedearlier than expected. If the packet is from one of the bondedcommunication links with the largest latency and its P-SEQ is equal tothe E-SEQ, all the packets in the queue with SEQ smaller than the P-SEQ,the packets, and packets with consecutive SEQ larger than the P-SEQ arethen delivered to a network interface of the DND, a device or a processaccording to order of the SEQs in order to deliver the packets insequence and reduce time-variance.

In one implementation, when packets are stored into the queue, eachpacket is assigned with a time tag to indicate a time for re-examinationof the packet. When the packet is re-examined, a decision is then madeto store the packet in the queue for a further period of time or todeliver the packet. If it is decided that the packet will be stored inthe queue for a further period of time, the time tag is then updated toa new value

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate embodiments of the invention and,together with the description, explain the invention. In the drawings,

FIG. 1A is a network diagram illustrating three bonded communicationlinks formed between one network interface of a source network devicewand three network interfaces of a destination network device, and thecorresponding exemplary matrix storing the latency differences among thethree bonded communication links,

FIG. 1B is a network diagram illustrating four bonded communicationlinks formed between two network interfaces of a source network deviceand two network interfaces of a destination network device, and thecorresponding exemplary matrix storing the latency differences among thefour bonded communication links,

FIG. 1C is a network diagram illustrating three bonded communicationlinks formed between three network interfaces of a source network deviceand one network interface of a destination network device, and thecorresponding exemplary matrix storing the latency differences among thethree bonded communication links,

FIG. 2 is a flow chart illustrating a method used to calculate thelatency differences,

FIG. 3 is a flow chart illustrating a method according to an embodimentof the present invention used to determine whether to deliver or tostore a packet, which is received from one of the bonded communicationlinks,

FIG. 4 is a flow chart illustrating a method according to an embodimentof the present invention used to determine whether to deliver or tostore a packet, which is received from one of the bonded communicationlinks, according to the sequence number of the packet and an expectedsequence number,

FIG. 5 is a flow chart illustrating a method according to an embodimentof the present invention used to determine whether to deliver or tostore a packet, which is received from one of the bonded communicationlinks, with a new time tag,

FIG. 6 is a flowchart illustrating a method according to an embodimentof the present invention, for processing packets, which have been storedin a queue of a destination network device, to further reduce timevariance when delivering packets,

FIG. 7 is a flowchart illustrating a method according to an embodimentof the present invention of delivering a packet,

FIG. 8 is a block diagram of a destination network device according toan embodiment of the present invention,

FIG. 9 is a block diagram of a destination network device according toan embodiment of the present invention with the use of a time tag and anexpected sequence number.

FIG. 10 is a flowchart illustrating a method according to an embodimentof the present invention, used to determine the total queue size basedon the sum of all the queues sizes.

MODE(S) FOR CARRYING OUT THE INVENTION Detailed Descriptions

Latency difference among bonded communication links is calculated bymeasuring the time difference of two packets, which are sentconsecutively from a source network device, arriving at a destinationnetwork device through two of the bonded communication links. As thesetwo packets arrive at the destination network device through twodifferent links, each packet may arrive at the destination networkdevice at different time due to different network conditions of thesetwo different links and the time the packets leaving the source networkdevice.

On the other hand, if consecutive packets are sent from the sourcenetwork device to the destination network device through one of thebonded communication links, it is assumed that there is no latencydifference between these consecutive packets because these two packetsshould experience similar network conditions.

As packets are continuously sent from the source network device to thedestination network device, latency difference may not remain constantbecause of changing network conditions. In order to reduce thepossibility of sudden change in latency difference, latency differencemay be calculated statistically, including using an exponential weightedmoving average algorithm to take into account of the past latencydifference and the current latency difference.

In order to allow a destination network device to identify the correctsequence of packets arriving from bonded communication links, it is acommon knowledge that source network device assigns a sequence number toeach packet. The sequence number may be embedded in the payload in anInternet Protocol packet, a payload in X.25 network, a TCP header, in anOSI model layer three packets, or any part of a packet. The destinationnetwork device may decapsulate a packet before processing the packet,storing the packet into a queue, and/or delivering the packet. It isapparent to a skilled person in the art that different encapsulation anddecapsulation methods and technologies may be used. It is also apparentto a skilled person in the art that when a packet is delivered by thedestination network device, the delivery may be implemented in manyways, including sending the packet to a network interface of thedestination network device, sending the packet to another deviceconnected to the destination network device, passing the packet to anapplication, a process or a thread running inside the destinationnetwork device, and storing the packet for another application.

FIG. 1A is a network diagram illustrating three bounded communicationlinks 110A, 110B and 110C connecting source network device 101 anddestination network device 104 from network interface 102 at sourcenetwork device 101 to three network interfaces 105, 106 and 107 atdestination network device 104 through interconnected network 103respectively. A source network device may include any device capable ofdistributing packets to one or bonded communication links, such asrouter, switch, mobile phone, multimedia device, and computer. The typeof a communication link may include any physical connection and/orlogical connection connecting a source network device and destinationnetwork device, such as a wireless connection, Ethernet connection,Internet Protocol connection, asynchronous transfer mode, virtualprivate network, WiFi, high-speed downlink packet access, GPRS, LTE andX.25, connecting a source network device and destination network device.A destination network device may include any device capable ofprocessing packets receiving from one or more links, such as a router,switch, mobile phone, multimedia device, or computer. For example, link110A may pass through a WiFi connection, link 110B and link 110C may usethe same type of GPRS transport but over two different serviceproviders. Interconnected network 103 includes Internet, intranet,private networks, public networks or combination of private and publicnetworks.

A memory 160 is used to store the time difference of packets withconsecutive sequence numbers arriving at destination network device 104from different links. Cell 161AA to cell 161CC are part of memory 160.The stored time difference in memory 160 may be used to estimate thelatency difference among different links. For example, a packet P1, witha sequence number one, originating from source network device 101 firsttravels through link 110A to arrive at destination network device 104.The next packet after P1, namely P2, originating from source networkdevice 101 travels from link 110B to arrive at destination networkdevice 104. The time difference between the arrivals of P1 and P2 atdestination network device 104 is then stored in cell 161AB of memory160 as P1 and P2 are packets with consecutive sequence numbers arrivingat destination network device 104 from link 110A and link 110Brespectively. Another example, if a packet with sequence number three,namely P3, arrives at destination network device 104 through link 110B,the time difference between P2 and P3 arriving at the destinationnetwork device 104 is not stored in memory 160 because P2 and P3 arriveat destination network device 104 through the same link. Similarly, if apacket with sequence number four, namely P4, arrives at the destinationnetwork device 104 through link 110A, the time difference between thearrivals of P3 and P4 at the destination network device 104 is stored incell 161BA of memory 160.

When packets from source network device 101 first arrive at destinationnetwork device 104, memory 160 may be empty. In order to calculate thetime differences among all links, for example, source network device 101may deliver packets with consecutive sequence numbers to destinationnetwork device 104 in the order of link 110A, link 110B, link 110C, link110B, link 110A, link 110C and link 110A and the arrival time of thepackets at destination network device 104 are recorded respectively. Thetime difference between packets' arrival time, in the order of arrival,which are the following sets: 110A and 110B, 110B and 110C, 110C, and110B, 110B and 110A, 110A and 110C, and finally 110C and 110A may thenbe stored in cell 161AB, cell 161BC, cell 161CB, cell 161BA, cell 161ACand cell 161CA respectively. When links are added or deleted betweensource network device 101 and destination network device 104, the valuestored in memory 160 may be reset to zero.

There is no value to be stored in cell 161AA, 161BB and 161CC because itis assumed that there is no latency difference for two consecutivepackets being sent to destination network device through the same link.

For example, if the latencies in links 110A, 110B and 110C are tenmilliseconds, twenty milliseconds and fifteen milliseconds respectively,and consecutive packets are sent from source network device 101 everyone milliseconds, the values in cell 161AA, 161AB, 161AC, 161BA, 161BB,161BC, 161CA, 161CB, and 161CC will then become null, eleven, six, minusnine, null, minus four, minus four, six and null respectively. When thefirst packet is sent from source network device 101 through 110A, thefirst packet may then arrive at destination network device 104 tenmilliseconds later. When the second packet is sent from source networkdevice 101 one millisecond later through 110B, the second packet maythen arrive at destination network device 104 twenty milliseconds later,or eleven seconds after the first packet's arrival at destinationnetwork device 104 because the latency difference between link 110A and110B is ten milliseconds and the second packet is sent one second afterthe first packet is sent from source network device 101. Therefore, thevalue in cell 161AB is eleven. Similarly, when the third packet is sentfrom source network device 101 through 110B, the third packet may thenarrive at destination network device 104 twenty milliseconds later. Whenthe fourth packet is sent from source network device 101 one millisecondlater through 110C, the packet may then arrive at destination networkdevice 104 fifteen milliseconds later, or four seconds earlier than thethird packet's arrival at destination network device 104 because thelatency difference between link 110B and 110C is minus five millisecondsand the fourth packet is sent one second after the third packet is sentfrom source network device 101. Therefore, the value in cell 161BC isminus four.

FIG. 1B is a network diagram illustrating four bonded communicationlinks 130A, 130B, 130C and 130D connecting source network device 121 anddestination network device 125 through network interfaces 122 and 123 atsource network device 121 and network interfaces 126 and 127 atdestination network device 125. A memory 170 is used to store the timedifference of two packets with consecutive sequence number arriving fromdifferent links.

FIG. 1C is a network diagram illustrating three bonded communicationlinks 150A, 150B, and 150C connecting source network device 141 anddestination network device 146 through network interfaces 142, 143 and144 at source network device 141 and network interfaces 147 atdestination network device 146. A memory 180 is used to store the timedifference of two packets with consecutive sequence numbers arrivingfrom different links.

FIG. 2 is a flow chart illustrating a method used to determine thedelay, based on latency difference determined, to be added to packetsarrived from links not with the largest latency. When latencydifferences among all links in a bonded communication links network aredetermined at functional block 201, the link with the largest latencycan then be determined at functional block 202. For example, as thelargest values of column A, column B and column C of memory 160 areeleven, minus four and six respectively, the link with the largestlatency is link 110B because column B has the smallest value among allthe columns.

The next step is to determine the amount of delay to be added to packetsarriving from different links at functional block 203. In oneembodiment, in order to reduce time variance when delivering packets,packets arrived through link 110A are delayed for eleven milliseconds,representing the largest cell value in column A and the sum of latencydifference and the time difference between two consecutive packetsleaving source network device 101. Similarly, packets arrived throughlink 110C are delayed for six milliseconds, representing the largestcell value in column C and the sum of latency difference and the timedifference between two consecutive packets leaving source network device101. However, for packets arriving through 110B, these packets aredelivered without delay because link 110B has the largest latency.

In one embodiment, the time difference between two packets withconsecutive sequence numbers arriving from two different links may becomputed with the values stored in memory 160 in order to update thevalues stored in memory 160. For example, the original value in cell161AB is eleven, which may indicate the sum of the latency differencebetween link 110A and link 110B and the time difference between twoconsecutive packets leaving source network device 101 was elevenmilliseconds, and the latency difference between the most recentlyreceived consecutive packets arriving from link 110A and 110B is twentymilliseconds, value in cell 161AB is then updated to a new valueaccording to an algorithm, for example exponential weighted movingaverage, in order to take into account of the recent twenty millisecondslatency difference experienced in link 110A and link 110B. It isapparent to a skilled person in the art that other algorithms may beused as well.

In one embodiment, when the time difference between two consecutivepackets sent from source network device 101 is unknown, destinationnetwork device 104 may treat the value stored in the cells of memory 160as latency difference, without taking into account of the timedifference between two consecutive packets sent from source

Memory 160, 170, and 180 may be implemented by using DRAM, SDRAM, FlashRAM, optical memory, magnetic memory, hard disk, and/or any othermaterials that are able to provide storage capability. The calculationof latency difference may be implemented by using one or more CPUs,ASICs, MCUs, microprocessors, and/or any devices that are able toprovide arithmetical functions.

When a packet has arrived at a destination network device, thedestination network device first determines which one of bondedcommunication links the packet has arrived from. If the packet hasarrived from a link with the largest latency, the packet is thendelivered. However, it is possible that there are other packets whichhave sequence numbers smaller than the sequence number of the packetalready being stored in a storage system of the destination networkdevice. These packets may have arrived at the destination network deviceearlier than the packet through other bonded communication links. Inorder to have in-sequence packet delivery, these packets are deliveredbefore the packet.

If the packet has arrived from a link not with the largest latency, thepacket may then be stored into a queue of a storage system of thedestination network device for later delivery. The period of the storagetime in the queue of the storage system of the destination networkdevice depends on latency difference in order to reduce time variancewhen delivering packets. The implementation of the queue and/or thestorage system may use DRAM, SDRAM, Flash RAM, optical memory, magneticmemory, hard disk, and/or any other materials that are able to providestorage capability.

Method

FIG. 3 is a flowchart illustrating a method for processing packetsreceived from bonded communication links according to the latencydifference among the bonded communication links and sequence numbers ofthe packets received. When a packet arrives at a destination networkdevice at functional block 301 through one of the bonded communicationlinks, the destination network device determines whether the packetarrives from the link with the largest latency at decision block 302. Ifthe packet arrives from the link with the largest latency, packets whichhave been stored in the queue earlier at functional block 305 withsequence numbers smaller than the sequence number of the packet will bedelivered at functional block 303 and followed by the delivery of thepacket at functional block 304. If the packet arrives not from the linkwith the largest latency, the packet is stored in a queue at functionalblock 305 for a period of time depending on the latency difference 306.

FIG. 4 is a flowchart illustrating a method for processing packetsreceived from bonded communication links according to the latencydifference among the bonded communication links, sequence numbers of thepackets received and an estimated sequence number.

Estimated sequence number may be used to predict what SEQ the nextpacket should be. Estimated sequence number may also be used to identifywhether a packet should be delivered if the sequence number of thepacket is compared differently to the estimated sequence number. Usingthis estimated sequence number in this invention assists thedetermination whether a particular packet inside the queue may bedelivered or the delay of the packet delay is being determinedaccurately.

In functional block 401, packet 451 arrives at a destination networkdevice through one of the bonded communication links. The sequencenumber of packet 451 is sequence number 452. In decision block 402, ifpacket 451 with sequence number 452 is less than the expected sequencenumber 453, packet 451 is then delivered in functional block 407 becausepacket 451 is considered arriving late. Alternatively, in decision block404, if packet 451 arrives from the link with the largest latency, it isthen delivered in functional block 407. If packet 451 arrives from alink other than the link with the largest latency, a time tag 454 atfunctional block 405 is then assigned to correspond to the period oftime that packet 451 is expected to be stored in the queue at functionalblock 406.

The value of a time tag is based on the latency difference. Using FIG. 1as an illustration for an implementation, if packet 451 arrives atdestination network device through link 110A and its sequence number 452is larger than expected sequence number 453, packet 451 is stored in thequeue and the value of time tag 454 is eleven because the largest valuein column A of memory 160 is eleven.

Once a packet has been stored in the queue, its associated time tag isexamined periodically to determine whether the packet should be examinedfor delivery. However, a packet may be delivered even before it is beingexamined or may continue to be stored in the queue after it is beingexamined if it is found that the latency difference estimation may notbe accurate or become outdate when network conditions of the bondedcommunication links change. In order to avoid a packet being stored forlonger than necessary when latency difference estimation is notaccurate, expected sequence number may be compared to the sequencenumber of the packet, and the value of the smallest sequence number ofthe packets stored in the queue may also be compared to the sequencenumber of the packet. A time limit threshold may also be used to preventthe packet has been stored in the queue too long. After a packet isremoved from the queue, the packet is then delivered. The value of thetime limit threshold may be determined by the destination networkdevice, entered by an administrator or pre-defined by the manufacturerof the destination network device.

FIG. 5 is a flowchart illustrating a method for processing a packetwhich has been stored in a queue of a destination network device. Thetime period for a packet staying in the queue may take into the accountof latency difference 306, time tag 454, and/or a pre-defined value.

In functional block 501, time tags of packets are periodically examined,for example for every five milliseconds, to identify packets which maybe ready for delivery. For example, when time tag 454, which is the timetag of packet 451, has indicated that packet 451 should be examined,functional block 501 identifies packet 451 for decision block 502. Indecision block 502, sequence number 452 is compared against expectedsequence number 453. If sequence number 452 is equal to expectedsequence number 453, it means that the estimation of latency differencemay still be accurate. Therefore, packet 451 is ready for delivery infunctional block 507.

If sequence number 452 is not equal to expected sequence number 453 indecision block 502, it may be an indication that the estimation oflatency difference may become inaccurate. Time tag 454 is examinedwhether packet 451 has been stored in the queue longer than a time limitthreshold in decision block 505. The time limit threshold may be anyvalue estimated by any device, selected by the device manufacturer, orinputted by a user of destination network device. According toexperimental results, the optimal value for time limit threshold for 3Gmobile link is in the range of seven hundred milliseconds to eighthundred milliseconds, whereas a typical ADSL or cable Ethernet link isin the range of two hundred and fifty milliseconds to three hundredmilliseconds. If packet 451 has been stored in the queue for a period oftime more than the time limit threshold in decision block 505.Therefore, packet 451 is ready for delivery in functional block 507.

If packet 451 has been stored in the queue for a period of time not morethan the time limit threshold in decision block 505, packet 451 may bestored in the queue for a further period of time. The value of time tag454 is then modified to a new value in functional block 506 that allowsto postpone the delivery of packet 451. The new value of time tag 454should allow packet 451 to be re-examined within a time period whichdoes not result in out-of-sequence delivery of packet 451. In oneembodiment, the new value of time tag 454 is set to be fivemilliseconds, such that packet 451 will then be re-examined fivemilliseconds later and latency difference estimation may then also beupdated.

In one embodiment, the step of decision block 502 is skipped. When apacket is examined, the only criterion to determine whether the packetshould be stored or delivered is whether the packet has been stored inthe queue for more than a time limit threshold in decision block 505.

FIG. 6 is a flowchart illustrating a method, based on the method shownin FIG. 5, for processing a packet which has been stored in a queue of adestination network device by taking into account of the sequencenumbers of packets stored in the queue. Decision block 601, functionalblock 602, functional block 603 and decision block 604 are added amongdecision block 505, functional block 506 and functional block 507 shownin FIG. 5. If sequence number 452 is not equal to expected sequencenumber 453 in decision block 502, the sequence number of the packet withthe lowest sequence number stored in the queue, for example packet 611,is compared against expected sequence number 453 at decision block 601.The sequence number and time tag of packet 611 are sequence number 612and time tag 613 respectively.

If sequence number 612 is equal to expected sequence number 453 indecision block 601, packet 611 is removed from the queue for delivery infunctional block 602. Further, expected sequence number 453 is increasedby one to indicate that one packet has been removed from the queue infunctional block 603. Expected sequence number 453 is then comparedagainst sequence number 452 in decision block 604. If expected sequencenumber 453 is equal to sequence number 452, it means that the estimationof latency difference is still valid. Therefore packet 451 is ready fordelivery in functional block 507.

If sequence number 612 is not equal to expected sequence number 453 indecision block 604, time tag 454 is examined whether packet 451 has beenstored in the queue for more than a time limit threshold in decisionblock 505. Steps to be performed at and after decision block 505 areidentical to the corresponding steps in FIG. 5.

In one embodiment, function block 601, function block 602, function 603and decision block 604 are visited only when sequence number 612 isfound to be equal to expected sequence number 453 at decision block 504for a predefined number of iterations, for example twice. Thisimplementation helps reducing the possibility for holding packets toolong in the queue when the estimation of latency difference becomesout-dated.

FIG. 7 is a flowchart illustrating a method of delivering a packet infunctional block 407 and functional block 507. Functional block 700provides functions identical to functional block 407 and functionalblock 507

When a packet is identified for delivery, there may be one or morepackets stored in the queue with sequence numbers smaller or larger thanthe sequence number of the packet. This may be due to a few reasons,including changing bonded communication links network environment,invalid latency estimation and packet loss. In one embodiment, in orderto reduce out-of-sequence packet delivery, if there is one or morepackets stored in the queue with sequence numbers smaller than thesequence number of the packet, these packets are delivered first infunctional block 701 and then followed by the delivery of the packet infunctional block 702. If there is one or more packets stored in thequeue with sequence numbers consecutively larger than the sequencenumber of the packet, these packets are delivered in block 703 after thepacket is delivered in block 702.

When a packet is delivered and its sequence number is larger thanexpected sequence number 453, expected sequence number 453 is updated tothe sequence number of the packet plus one to indicate the sequencenumber of the next packet expected to be delivered. When more than onepacket are delivered, expected sequence number 453 is updated to thelargest sequence number of the packets plus one to indicate that thesequence number of the next packet expected to be delivered.

System

A system may have one or more ingress interfaces for receiving packetsand one or more egress interfaces for sending packets. An interface maybe able to perform both roles of ingress interface and egress interface.A system may also have one or more control modules. For example, onecontrol module is responsible for network interface and one controlmodule is responsible for data storage system. The control modules maycommunicate among themselves. It is also possible that one controlmodule is responsible for all control mechanisms in the system. It isapparent to a skilled person in the art that one or more control modulescan be implemented in many variations.

FIG. 8 is a block diagram illustrating a system for processing packetsreceived from bonded communication links according to the latencydifference among the bonded communication links and sequence numbers ofthe packets received. Control module 803 may be a single control module,may be composed of multiple control modules or may include one or morecontrol modules. Control module may be comprised of one or more CPUs,ASICs, MCUs, microprocessors, and/or any devices that are able toprovide control functionalities. For example, to calculate latencydifference, control module 803 compares the difference in arrival timeof two packets which have consecutive sequence numbers arriving from twodifferent links to estimate the latency difference among differentlinks. The estimated latency differences may then be stored at storagesystem 804.

When a packet arrives at one of the ingress interfaces 801, controlmodule 803 determines whether the packet arrived is from the link withthe largest latency. If the packet is from the link with the largestlatency, the packet should then be sent to one of the egress interfaces802 depending on the destination of the packet. If the packet is notfrom the link with the largest latency, the packet should then be storedin queue 805 of storage system 804 for later delivery because the packetis assumed to be arriving earlier than other packets. Storage system 804may be implemented by using DRAM, SDRAM, Flash RAM, optical memory,magnetic memory, hard disk, and/or any other materials that are able toprovide storage capability. Queue 805 may be a section in storage system804 or the whole of storage system 804.

In one embodiment, based on the sequence number of last packet deliveredto egress interface 802, control module 803 determines the value ofexpected sequence number. For example, if the sequence number of lastpacket delivered to egress interface 802 is thirty-three, control module803 may update the expected sequence number to be thirty-four toindicate the sequence number of next packet to be sent is expected to bethirty-four. Control module 803 compares the sequence number of a packetarrived from one of the ingress interfaces 801 against the expectedsequence number. If the sequence number of the packet arrived is smallerthan the expected sequence number, the packet is delivered without beingstored in queue 805 because it is assumed the packet has arrived laterthan expected. If the sequence number of the packet arrived is notsmaller than the expected sequence number and the packet is from thelink with the largest latency, the packet should then be delivered toone of the egress interfaces 802 depending on the destination of thepacket. On the other hand, if the sequence number of the packet arrivedis not smaller than the expected sequence number and the packet is notfrom the link with the largest latency, the packet should then be storedin queue 805 of storage system 804 for later delivery because the packetis assumed to be arriving earlier than other packets.

FIG. 9 is an embodiment to illustrate how a system process packets thathave been stored in a queue. When control module 803 stores a packet inqueue 805, control module 803 stores the time when the packet is goingto be examined again in time tag 806. The value of time tag 806 is basedon latency difference. The time period for a packet staying in queue 805may take into the account of latency difference, time tag 806, and/or apre-defined value. Time tag 806 may be implemented by using DRAM, SDRAM,SRAM, or FLASH RAM placed inside control module 803 and/or part ofstorage system 804.

Control module 803 may periodically, for example for every fivemilliseconds, examine queue 805 to identify packets which may be readyfor delivery. Control module 803 may also be alerted by time tag 806 forpacket which may be ready for delivery.

For example, when packet 808 is identified for the possibility ofdelivery, control module 803 compares the sequence number 809 of packet808 against expected sequence number 807. If the sequence number 809 isequal to expected sequence number 807, it means that the estimation oflatency may still be accurate. Therefore control module 803 may sendpacket 808 to egress interface 802 for delivery.

If sequence number 809 is not equal to expected sequence number 807, itmay be an indication that the estimation of latency difference maybecome inaccurate. Control module 803 then examines time tag 806 todetermine whether packet 808 has been stored in queue 805 longer thanthe time limit threshold. Control module 803 delivers packet 808 ifpacket 808 has been stored in queue 805 for more than the time limitthreshold. On the other hand, control module 803 may store packet 808 inqueue 805 for a further period of time if packet 808 has not been storedin queue 805 for more than the time limit threshold. Control module 803then modifies the value of time tag 806 to a new value that allowspostponing the delivery of packet 808. The new value of time tag 806should allow packet 808 to be re-examined by control module 803 within atime period which does not result in out-of-sequence delivery of packet808. In one embodiment, the new value of time tag 806 is set to be fivemilliseconds, such that packet 808 will then be re-examined fivemilliseconds later and latency difference estimation may then also beupdated. The time limit threshold can be any value estimated by controlmodule 803, any device, selected by the device manufacturer, or inputtedby a user of destination network device. According to experimentalresults, the optimal value for time limit threshold for 3G mobile linkis in the range of seven hundred milliseconds to eight hundredmilliseconds, whereas a typical ADSL or cable Ethernet link is in therange of two hundred and fifty milliseconds to three hundredmilliseconds.

Control module 803 may determine, based on the sequence number of lastpacket delivered to egress interface 802, the value of expected sequencenumber 807. For example, if the sequence number of last packet deliveredto egress interface 802 is thirty-three, control module 803 may updateexpected sequence number 807 to be thirty-four to indicate that thesequence number of next packet to be sent is expected to be thirty-four.

In one embodiment, control module 803 may determine whether the packetshould be stored further in queue 805 or delivered to egress interface802 solely based on whether the packet has been stored in queue 805 formore than the time limit threshold.

In one embodiment control module 803 takes into account of the sequencenumbers of packets stored in queue 805 when processing packets. Whencontrol module 803 identifies a packet, for example packet 808, for thepossibility of delivery, control module 803 compares the sequence number809 of packet 808 against expected sequence number 807. If the sequencenumber 809 is equal to expected sequence number 807, control module 803may send packet 808 to egress interface 802 for delivery.

If the sequence number 809 is not equal to expected sequence number 807,control module 803 then compares the lowest sequence number of thepacket stored in queue against expected sequence number 807, for examplesequence number 811 of packet 810. Control module 803 identifies packet810 by, for example, examining the sequence numbers of all the packetsstored in queue 805. If queue 805 is a sorted queue by sequence number,packet 810 may be placed at the top or bottom of queue 805

If control module 803 determines that sequence number 811 is equal toexpected sequence number 807, control module 803 removes packet 810 fromqueue 805 to egress interface 802 for delivery. Further, control module803 increases expected sequence number 807 by one to indicate that onepacket has been removed from queue 805. Control module 803 then comparesexpected sequence number 807 against sequence number 809. If expectedsequence number 809 is equal to sequence number 807, it means that theestimation of latency difference is still valid. Therefore, controlmodule 803 removes packet 808 from queue 805 to egress interface 802 fordelivery.

If sequence number 809 is not equal to expected sequence number 807,control module 803 then examines time tag 806 to determine whetherpacket 808 has been stored in queue 805 for more than a time limitthreshold. If control module 803 determines that packet 808 has beenstored in queue 805 for a period of time more than the time limitthreshold, control module 803 retrieves packet 808 from queue 805 anddeliver packet 808 to egress interface 802 for delivery. If packet 808has been stored in the queue for a period of time not more than the timelimit threshold, packet 808 may be stored in the queue for a furtherperiod of time. Control module 803 updates the value of time tag 806 toa new value that allows postponing the delivery of packet 808. The newvalue of time tag 806 should allow packet 808 to be re-examined within aperiod of time which does not result in out-of-sequence delivery ofpacket 808. In order to reduce the out-sequence packet delivery, in oneembodiment, the new value of time tag 808 is set to be five millisecondslater.

In one embodiment, before control module 803 sends packet 808 to egressinterface 802 for delivery, control module 803 checks if there are oneor more packets stored in queue 805 with sequence numbers smaller orlarger than the sequence number of the packet 808, these packets aresent to egress interface 802 first and then followed by the packet 808.If there are one or more packets stored in queue 805 with sequencenumber consecutively larger than the sequence number of packet 808,control module 803 sends these packets to egress interface 802 afterpacket 808.

In one embodiment, control module 803 updates expected sequence number807 to be the sequence number of the packet just being sent to egressinterface 802 plus one to indicate that the sequence number of the nextpacket expected to be sent to egress interface 802.

FIG. 10 illustrates a process to determine the total queue size requiredaccording to one of embodiments of the present invention. As a queue isrequired to store packets, it is preferred to have a queue that is largeenough to store packets but not too large that consumes unnecessaryresources and resulting in some of the queue not used.

In step 1101, the latency of the link with the largest latency isestimated. For readability, the latency of the link with the largestlatency is referred to be Largest Latency. Those who are skilled in theart would appreciate that there are myriad ways of estimating latency,such as using ping command.

In step 1102, for each link in the bonded communication links, the queuesize of the link is determined. The queue size is determinedsubstantially based on the Largest Latency and the packets arrival speedof the particular link. Using FIG. 1A for illustration purpose, thelatencies in links 110A, 110B and 110C are ten milliseconds, twentymilliseconds and fifteen milliseconds respectively. Bandwidth of links110A, 110B and 110C are 30 Mbps, 20 Mbps and 10 Mbps. As link 110B hasthe largest latency of twenty milliseconds, Largest Latency is twentymilliseconds. As a result, the queue size for link 110A will be, 30 Mbpstimes twenty milliseconds, 75 kilobytes; the queue size for link 110Cwill be, 10 Mbps times twenty milliseconds, 25 kilobytes. There is noneed to have a queue for link 110B as packets arrived from link 110Bwill be forwarded when the packets arrive.

In step 1120, the total queue size is the sum of all the queue sizes ofall the links in the bonded communication links, excluding the link withthe largest latency. Therefore, using the same illustration, the totalqueue size is the sum of 75 kilobytes and 25 kilobytes and is 100kilobytes.

In one variance, in order to anticipate early arrival of packets fromlink 110B, a queue is also required for link 110B. The queue size of thequeue for link 110B is preferred to be one quarter to one half ofLargest Latency times its bandwidth. The queue size for link 110B atstep 1102 therefore is in the range of 12.5 kilobytes (20 Mbps timesfive milliseconds) and 25 kilobytes (20 M times five milliseconds). Thetotal queue size becomes 125 kilobytes in step 1120.

In one variance, all the links share one common queue and the queue sizeat step 1102 is flexible to store packets, which are arrived from lowerlatency links, that has not been stored longer than the Largest Latency.When a packet has been stored longer than the Largest Latency, thepacket will be discarded in order to preserve storage of the queue. Inone variance, each link has its own queue and the size of each queue isnot fixed.

In one variance, packets are allowed to be stored for a time durationthat is longer than the Largest Latency as long as the total queue sizeis not more than a predefined value. When the pre-defined queue size isreached, packets that have been stored the longest will be discarded.This allows more packets to be stored while not cause unexpected amountof storage being used for the queue. There are advantages anddisadvantages to allow each link has its own queue when comparing to useone common queue.

In the case of allowing each link has its own queue, finer configurationcan be achieved. For example, using the same illustration of FIG. 1A,the maximum queue size of link 110A is set to 1000 kilobytes and thetime duration limit for packets to be stored is one second. Further, forlinks 110B and 110C, the maximum queue sizes and time duration limitscan be configured individually. This reduce the probability that anunexpected rise of latency in one link or a sudden increase in bandwidthof one link will consume most of the available queue storage.

In the case of using one common queue, the common queue allows thebenefits of statistical multiplexing and accommodate larger variance oflatency and bandwidth of each link of the bonded communication links.For example, the total queue size for the common queue is set to fivemegabytes. In one variance, further, the allowed time duration of packetstorage is set to five seconds. Therefore, when a packet arrives, if itis not being sent immediately, it will be stored in the common queue. Ifthere is no storage left, packets that have been stored the longest willbe discarded in order to create storage space for the newly arrivedpacket. In one variance, the size of the common queue is not based on apredefined value. Instead, the size of the common queue is based on theLargest Latency and bandwidth of each link of the bonded communicationlinks.

INDUSTRIAL APPLICABILITY

This invention relates in general to network communications and, moreparticularly, to a method and system for processing packets receivedfrom bonded communication links according to latency difference amongthe bonded communication links and sequence numbers. Network trafficreceived from bonded communication links are delivered to a device, anetwork interface or a process of a destination network device insequence with higher probability and less time variance comparing to adestination network device without implementing this invention.

1. A method for storing packets received from a bonded communicationlinks, wherein the bonded communication links comprising of a pluralityof communication links, comprising the steps of: (a) determining latencyof the link with largest latency; (b) estimating storage size of aqueue, wherein the queue is for storage the one or more packets; (c)when receiving one or more packets from the bonded communication links,determining whether to store the one or more packets in the queue or toforward the one or more packets; and wherein the one or more packetsreceived contain sequence numbers indicating a sequence of the one ormore packets sent from a source network device; wherein the one or morepackets are sent from the source network device without any particularorder of sending through the bonded communication links.
 2. The methodof claim 1, wherein the storage size is estimated based on the largestlatency and bandwidth of each of the plurality of communication links.3. The method of claim 1, wherein the queue is comprised of a pluralityof individual queues, wherein each of the plurality of individual queuescorresponds to one communication link of the plurality of communicationlinks.
 4. The method of claim 1, wherein the storage size is estimatedbased on the sum of bandwidth times latency of each of the plurality ofcommunication links.
 5. The method of claim 1, wherein the storage sizeis based on an allowed time duration of packet storage.
 6. The method ofclaim 5, wherein the time duration is configurable.
 7. The method ofclaim 5, wherein the time duration is based on the largest latency. 8.The method of claim 7, wherein the storage size is not larger than apre-configured size.
 9. The method of claim 8, further comprisingdiscarding packets that have been stored the longest when there isinadequate storage.
 10. The method of claim 1, further comprisingre-determining the latency of the link with largest latency differencewhen one or more communication links are added to or deleted from thebonded communication links.
 11. A system for storing packets receivedfrom a bonded communication links, wherein the bonded communicationlinks comprising of a plurality of communication links, comprising: oneor more network interfaces for receiving one or more packets from thebonded communication links, and one or more control modules areconfigured for: (a) determining latency of the link with largestlatency; (b) estimating storage size of a queue, wherein the queue isfor storage the one or more packets; when receiving one or more packetsfrom the bonded communication links, determining whether to store theone or more packets in the queue or to forward the one or more packets;and wherein the one or more packets received contain sequence numbersindicating a sequence of the one or more packets sent from a sourcenetwork device; wherein the one or more packets are sent from the sourcenetwork device without any particular order of sending through thebonded communication links.
 12. The system of claim 11, wherein thestorage size is estimated based on the largest latency and bandwidth ofeach of the plurality of communication links.
 13. The system of claim11, wherein the queue is comprised of a plurality of individual queues,wherein each of the plurality of individual queues corresponds to onecommunication link of the plurality of communication links.
 14. Thesystem of claim 11, wherein the storage size is estimated based on thesum of bandwidth times latency of each of the plurality of communicationlinks.
 15. The system of claim 11, wherein the storage size is based onan allowed time duration of packet storage.
 16. The system of claim 15,wherein the time duration is configurable.
 17. The system of claim 15,wherein the time duration is based on the largest latency.
 18. Thesystem of claim 17, wherein the storage size is not larger than apre-configured size.
 19. The system of claim 18, wherein the one or morecontrol modules are further configured for: discarding packets that havebeen stored the longest when there is inadequate storage.
 20. The systemof claim 11, wherein the one or more control modules are furtherconfigured for redetermining the latency of the link with largestlatency difference when one or more communication links are added to ordeleted from the bonded communication links.